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ABSTRACT 



One of the mandates of the Kentucky Education Reform Act of 
1990 is that students develop a writing portfolio. Ultimately, schools must 
elevate the average performance level of students’ portfolios to the 
benchmark of "proficient." During site visits to 29 schools, 34 indicators 
were identified that differentiated writing scores and programs in 
continuously improving schools from those in continuously declining schools. 

A self-study needs assessment instrument-- the School Study of Writing 
Instruction- -was developed for schools to assess their needs relative to the 
indicators. The instrument has four operational stages: conducting interviews 
with teachers, students, and administrators; writing a report framed around 
the 34 indicators; using the report in a faculty-wide session to rate the 
school on the indicators; and using the ratings to set priorities and plan 
actions for improvement of the writing program. This report describes the 
pilot testing of the instrument in one small Kentucky elementary school and 
the testing of parts of it by a few other entities. The pilot test assessed 
the value of the self-study for schools, the level of external facilitator 
assistance needed by schools, and the validity and reliability of the 
instrument. The self-study was successful for the pilot- test school, which 
was attributed to a faculty sense of ownership and to the school ' s 
customizing of the instrument. The whole process was completed in 5 weeks. 
Facilitator assistance was critical but was not needed constantly. Revisions 
are proposed to minimize threats to the instrument's validity and 
reliability. Appendices include parts of the instrument and evaluation 
materials. (Contains 21 references.) (SV) 
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EXECUTIVE SUMMARY 



One of the mandates of the Kentucky Education Reform Act (KERA) of 1 990 is that students 
develop a writing portfolio (Kentucky Department of Education, 1990, 1998). Ultimately, schools 
in Kentucky are expected to elevate the average of students’ portfolios to the benchmark of 
proficient. To aid in this endeavor, the Kentucky Department of Education (KDE) and Appalachia 
Educational Laboratory (AEL), Inc. began a collaborative project in 1996 (AEL, 1997a). The 
original goal of the project was to design new professional development programs for teachers in the 
area of writing instruction. 

The project proposal included a preliminary telephone survey to assess teachers’ professional 
development needs prior to designing new programs (AEL, 1995). However, at the first meeting 
between KDE and AEL, key decision makers realized that a more in-depth scope for background 
research was in order. They decided against the telephone survey, which would only be useful for 
gaining superficial insights, in favor of a series of site visits that would more systematically ascertain 
teachers’ needs. More specifically, the observations made during site visits would establish 
indicators of quality writing programs and thereby determine needs of less successful schools by 
their apparent lack of these indicators. At the second meeting, a collaborative research team 
composed of KDE and AEL staff was formed. Through site visits to 29 schools, the research team 
identified 34 indicators that differentiated writing scores and programs in continuously improving 
schools from those in continuously declining schools, of which all but one attained statistical 
significance. These indicators surpass the scope of professional development alone to address issues 
such as instructional strategies, administrative support, and family and community involvement. 

Next, the collaborative research team evaluated the optimum method for assessing schools’ 
needs relative to the indicators, deciding upon a self-study needs assessment instrument. The 
distinguishing characteristics of the study are that (1) it is to be conducted by schools themselves (led 
by an in-house faculty-based Steering Committee rather than a team of external evaluators) and (2) 
it has the capability to account for the perspectives of all role groups in the school in a compelling 
way, including students’ perspectives. The study’s power is principally derived from comparing the 
consistencies and inconsistencies among the different role groups’ perspectives about the writing 
portfolio program. 

The instrument is comprised of four operational elements, or steps, compiled in a single 
manual for the school’s guidance. The four steps are (1) conducting interviews, (2) writing a report, 
(3) rating the school on the 34 indicators, and (4) setting priorities. The Steering Committee plans 
and carries out most of the tasks in the process except the interviews. The four role groups to be 
interviewed are teachers, students, the principal, and a district-level administrator. Based on the 
interview data, the Committee writes a report framed around the 34 indicators of successful writing 
programs. These data compilation steps culminate in a faculty-wide rating session led by the 
Committee. In the faculty-wide rating session, small groups use the report to rate the school on the 
indicators along a 7-point Likert-type scale. Eventually, the ratings are used to set priorities and plan 
actions for improvement of the writing program. 
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Describing the pilot test of this new needs assessment instrument, called the School Study 
of Writing Instruction (AEL & KDE, 1998), is the purpose of this report. The objectives of the pilot 
test were to assess the value of the study for schools, assess the level of external facilitator assistance 
needed by schools to carry it out, and assess the validity and reliability of the instrument itself. The 
School Study of Writing Instruction was pilot tested by two different methods. First and foremost, 
it was tested in its entirety in a small Kentucky elementary school (Method I, the full model). The 
pilot-test school was assigned an external observer from the collaborative research team, whose role 
evolved into one more closely resembling a facilitator. This development was advantageous in 
hindsight as it resulted in more complete findings from the pilot test. Second, the instrument was 
tested in sections by six educators around the state (Method II, the sectional model). 

The self-study was successful for the pilot-test school in Method I. One observed outcome 
was the Steering Committee’s practice of customizing the instrument while conducting the study, 
a noteworthy example of Berman and McLaughlin’s (1975) concept of “mutual adaptation” in 
education innovation implementation (Hord, 1987). The facilitator attributed the school’s success 
to a faculty sense of ownership of the program generated by faculty participation at the rating 
meeting. Moreover, the study process was efficient for the general faculty body; however, the 
compromise was that the process was time-intensive for the Steering Committee, as Committee 
members had been thorough in their planning and preparation. Yet overall, the process was 
parsimonious in that all steps up to, but not including, step 4 (the priorities) were accomplished in 
five weeks. Findings from Method II were also informative, though not as comprehensive as those 
from Method I. Participants in Method II, who tested sections of the instrument, were positive about 
the study’s potential but concerned about the required time commitment. 

The observer/facilitator’s assistance was judged to be critical, but not constantly needed. In 
her account, her best service was pointing out the essential pages of the study manual to the Steering 
Committee at their initial planning meeting, thereby making the study process seem manageable and 
straightforward. It is recommended that schools undertaking the School Study in the future have 
access to some level of assistance, whether it is provided by in-house or external facilitators. 

In Method I, a validity concern was identified in regard to transposing the interview data to 
the report, as well as other minor problems in the instrument. Although the Steering Committee’s 
report captured essential highlights of the school’s writing program, it seemed too brief. Revisions 
are proposed to the instrument to minimize threats to validity and reliability. Nevertheless, any 
residual invalidity and unreliability pale in significance to the high value of the study for faculty. 
It is more important for a faculty to be empowered by the study process than too concerned with the 
mechanics of it. The School Study of Writing Instruction (AEL & KDE, 1 998) is designed to build 
capacity of schools to assess their own progress with respect to writing improvement. In this pilot 
test, the potential of the School Study to enable a school faculty to own the problems, data, and 
solutions related to its writing program was adequately demonstrated. It is recommended that the 
School Study be marketed nationwide to educators who wish to improve writing programs. Also, 
it is recommended that the School Study, as a model needs assessment instrument, be adapted to 
other content areas and marketed likewise. 
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INTRODUCTION 



One of the mandates of the Kentucky Education Reform Act (KERA) of 1990 is that every 
student develop a writing portfolio for which he/she is accountable at the 4 lh , 7 th , and 12 lh grades 
(Kentucky Department of Education, 1994b, 1996, 1999). There are four benchmarks of 
achievement by which writing portfolios are judged: novice, apprentice, proficient, and 

distinguished. Schools are required by KERA to achieve an average of proficient on students’ 
writing portfolios by 2010. However, since the inception of KERA, teachers have found it possible 
to move students from novice to apprentice, but more formidable to maneuver them to the next 
benchmark of proficient. Presently, many students at varying degrees of mastery have been 
appraised at the apprentice level. (Some teachers have voiced concern that the range of mastery 
within the apprentice benchmark is too wide. They feel that many students deserve to be evaluated 
at a higher level, and to adequately accommodate their range, intermediate benchmarks between 
apprentice and proficient should be created.) In 1996, the Kentucky Department of Education 
(KDE) and Appalachia Educational Laboratory (AEL), Inc. began a collaborative project to assist 
teachers in writing instruction and ultimately, to improve their students’ writing portfolio scores 
(AEL, 1995, 1997a). The project, titled the Kentucky State Project: Designing Professional 
Development for Portfolio Improvement, shall be hereafter referred to as the Kentucky writing 
project or project. 

Prior to the inception of the project, approximately 100 Kentucky schools were identified 
as having shown consistent gains in writing portfolio scores between 1991-1994 (AEL, 1995). 
Furthermore, although some have criticized the writing portfolio scoring process as being subjective, 
scoring audits showed that scores in these schools were more reliable than scores in less successful 
schools. To help less successful schools achieve these same gains, AEL and KDE were interested 
in a preliminary study to identify correlates of score gains in successful schools, possibly followed 
by the creation of new professional development materials and strategies (AEL, 1 995). As stated 
in the project proposal, the original objectives of the research and development effort were (1) to 
establish correlates to gains in writing scores; (2) based on these correlates, to develop and field test 
new professional development materials to improve both teacher writing instruction and accuracy 
in scoring student portfolios, implement the new materials in a training program, and disseminate 
the materials statewide and nationally; and (3) to determine the applicability of the new professional 
development materials to content areas other than writing instruction. 

The Kentucky writing project was conceived as an applied research and development effort. 
In the first meeting between KDE and AEL, the original objectives and activities of the project 
changed substantially (AEL, 1998a). Focus converged on the first objective, that of identifying 
correlates of writing score gains, later termed quality indicators or indicators. The second and third 
objectives of designing new professional development were put on hold, because key decision 
makers at KDE agreed that preliminary research into correlates of gains required in-depth attention. 
First, KDE staff proposed that advancing students to the proficient level requires a completely 
different set of teaching strategies than those that had been sufficient for getting students to the 
apprentice level (AEL, 1998a). Second, KDE hypothesized that perhaps other variables in addition 
to professional development and teaching strategies were decisive factors in successful schools’ 
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